{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Dropout\n",
    "Dropout [1] is a technique for regularizing neural networks by randomly setting some features to zero during the forward pass. In this exercise you will implement a dropout layer and modify your fully-connected network to optionally use dropout.\n",
    "\n",
    "[1] Geoffrey E. Hinton et al, \"Improving neural networks by preventing co-adaptation of feature detectors\", arXiv 2012"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting future\n",
      "Installing collected packages: future\n",
      "Successfully installed future-0.17.1\n"
     ]
    }
   ],
   "source": [
    "!pip install future"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "run the following from the cs231n directory and try again:\n",
      "python setup.py build_ext --inplace\n",
      "You may also need to restart your iPython kernel\n"
     ]
    }
   ],
   "source": [
    "# As usual, a bit of setup\n",
    "from __future__ import print_function\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from cs231n.classifiers.fc_net import *\n",
    "from cs231n.data_utils import get_CIFAR10_data\n",
    "from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array\n",
    "from cs231n.solver import Solver\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for auto-reloading external modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "def rel_error(x, y):\n",
    "  \"\"\" returns relative error \"\"\"\n",
    "  return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train:  (49000, 3, 32, 32)\n",
      "y_train:  (49000,)\n",
      "X_val:  (1000, 3, 32, 32)\n",
      "y_val:  (1000,)\n",
      "X_test:  (1000, 3, 32, 32)\n",
      "y_test:  (1000,)\n"
     ]
    }
   ],
   "source": [
    "# Load the (preprocessed) CIFAR10 data.\n",
    "\n",
    "data = get_CIFAR10_data()\n",
    "for k, v in data.items():\n",
    "  print('%s: ' % k, v.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Dropout forward pass\n",
    "In the file `cs231n/layers.py`, implement the forward pass for dropout. Since dropout behaves differently during training and testing, make sure to implement the operation for both modes.\n",
    "\n",
    "Once you have done so, run the cell below to test your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running tests with p =  0.3\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  6.989686039362353\n",
      "Mean of test-time output:  3.00006236354325\n",
      "Fraction of train-time output set to zero:  0.300876\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.6\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  4.014061423217988\n",
      "Mean of test-time output:  6.0001247270865\n",
      "Fraction of train-time output set to zero:  0.598632\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.75\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  2.5054064372816414\n",
      "Mean of test-time output:  7.500155908858126\n",
      "Fraction of train-time output set to zero:  0.749504\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(500, 500) + 10\n",
    "\n",
    "for p in [0.3, 0.6, 0.75]:\n",
    "  out, _ = dropout_forward(x, {'mode': 'train', 'p': p})\n",
    "  out_test, _ = dropout_forward(x, {'mode': 'test', 'p': p})\n",
    "\n",
    "  print('Running tests with p = ', p)\n",
    "  print('Mean of input: ', x.mean())\n",
    "  print('Mean of train-time output: ', out.mean())\n",
    "  print('Mean of test-time output: ', out_test.mean())\n",
    "  print('Fraction of train-time output set to zero: ', (out == 0).mean())\n",
    "  print('Fraction of test-time output set to zero: ', (out_test == 0).mean())\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Dropout backward pass\n",
    "In the file `cs231n/layers.py`, implement the backward pass for dropout. After doing so, run the following cell to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dx relative error:  1.8928951796143967e-11\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(10, 10) + 10\n",
    "dout = np.random.randn(*x.shape)\n",
    "\n",
    "dropout_param = {'mode': 'train', 'p': 0.8, 'seed': 123}\n",
    "out, cache = dropout_forward(x, dropout_param)\n",
    "dx = dropout_backward(dout, cache)\n",
    "dx_num = eval_numerical_gradient_array(lambda xx: dropout_forward(xx, dropout_param)[0], x, dout)\n",
    "\n",
    "print('dx relative error: ', rel_error(dx, dx_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Fully-connected nets with Dropout\n",
    "In the file `cs231n/classifiers/fc_net.py`, modify your implementation to use dropout. Specificially, if the constructor the the net receives a nonzero value for the `dropout` parameter, then the net should add dropout immediately after every ReLU nonlinearity. After doing so, run the following to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running check with dropout =  0\n",
      "Initial loss:  2.3004790897684924\n",
      "W1 relative error: 1.48e-07\n",
      "W2 relative error: 2.21e-05\n",
      "W3 relative error: 3.53e-07\n",
      "b1 relative error: 5.38e-09\n",
      "b2 relative error: 2.09e-09\n",
      "b3 relative error: 5.80e-11\n",
      "\n",
      "Running check with dropout =  0.25\n",
      "Initial loss:  2.3038652786350595\n",
      "W1 relative error: 2.60e-07\n",
      "W2 relative error: 3.22e-06\n",
      "W3 relative error: 1.78e-06\n",
      "b1 relative error: 1.24e-08\n",
      "b2 relative error: 2.13e-09\n",
      "b3 relative error: 1.33e-10\n",
      "\n",
      "Running check with dropout =  0.5\n",
      "Initial loss:  2.304470252186846\n",
      "W1 relative error: 9.14e-08\n",
      "W2 relative error: 1.13e-07\n",
      "W3 relative error: 2.59e-07\n",
      "b1 relative error: 1.83e-08\n",
      "b2 relative error: 3.38e-09\n",
      "b3 relative error: 8.35e-11\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "N, D, H1, H2, C = 2, 15, 20, 30, 10\n",
    "X = np.random.randn(N, D)\n",
    "y = np.random.randint(C, size=(N,))\n",
    "\n",
    "for dropout in [0, 0.25, 0.5]:\n",
    "  print('Running check with dropout = ', dropout)\n",
    "  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,\n",
    "                            weight_scale=5e-2, dtype=np.float64,\n",
    "                            dropout=dropout, seed=123)\n",
    "\n",
    "  loss, grads = model.loss(X, y)\n",
    "  print('Initial loss: ', loss)\n",
    "\n",
    "  for name in sorted(grads):\n",
    "    f = lambda _: model.loss(X, y)[0]\n",
    "    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)\n",
    "    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Regularization experiment\n",
    "As an experiment, we will train a pair of two-layer networks on 500 training examples: one will use no dropout, and one will use a dropout probability of 0.75. We will then visualize the training and validation accuracies of the two networks over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "(Iteration 1 / 125) loss: 7.856643\n",
      "(Epoch 0 / 25) train acc: 0.274000; val_acc: 0.192000\n",
      "(Epoch 1 / 25) train acc: 0.410000; val_acc: 0.263000\n",
      "(Epoch 2 / 25) train acc: 0.518000; val_acc: 0.269000\n",
      "(Epoch 3 / 25) train acc: 0.550000; val_acc: 0.248000\n",
      "(Epoch 4 / 25) train acc: 0.684000; val_acc: 0.297000\n",
      "(Epoch 5 / 25) train acc: 0.758000; val_acc: 0.292000\n",
      "(Epoch 6 / 25) train acc: 0.782000; val_acc: 0.266000\n",
      "(Epoch 7 / 25) train acc: 0.864000; val_acc: 0.241000\n",
      "(Epoch 8 / 25) train acc: 0.860000; val_acc: 0.283000\n",
      "(Epoch 9 / 25) train acc: 0.892000; val_acc: 0.281000\n",
      "(Epoch 10 / 25) train acc: 0.910000; val_acc: 0.266000\n",
      "(Epoch 11 / 25) train acc: 0.944000; val_acc: 0.285000\n",
      "(Epoch 12 / 25) train acc: 0.968000; val_acc: 0.305000\n",
      "(Epoch 13 / 25) train acc: 0.972000; val_acc: 0.306000\n",
      "(Epoch 14 / 25) train acc: 0.962000; val_acc: 0.280000\n",
      "(Epoch 15 / 25) train acc: 0.994000; val_acc: 0.290000\n",
      "(Epoch 16 / 25) train acc: 0.984000; val_acc: 0.309000\n",
      "(Epoch 17 / 25) train acc: 1.000000; val_acc: 0.310000\n",
      "(Epoch 18 / 25) train acc: 0.976000; val_acc: 0.296000\n",
      "(Epoch 19 / 25) train acc: 0.928000; val_acc: 0.308000\n",
      "(Epoch 20 / 25) train acc: 0.990000; val_acc: 0.305000\n",
      "(Iteration 101 / 125) loss: 0.001978\n",
      "(Epoch 21 / 25) train acc: 0.970000; val_acc: 0.316000\n",
      "(Epoch 22 / 25) train acc: 0.972000; val_acc: 0.323000\n",
      "(Epoch 23 / 25) train acc: 0.978000; val_acc: 0.297000\n",
      "(Epoch 24 / 25) train acc: 0.970000; val_acc: 0.289000\n",
      "(Epoch 25 / 25) train acc: 0.978000; val_acc: 0.290000\n",
      "0.75\n",
      "(Iteration 1 / 125) loss: 5.308521\n",
      "(Epoch 0 / 25) train acc: 0.258000; val_acc: 0.190000\n",
      "(Epoch 1 / 25) train acc: 0.428000; val_acc: 0.252000\n",
      "(Epoch 2 / 25) train acc: 0.478000; val_acc: 0.284000\n",
      "(Epoch 3 / 25) train acc: 0.532000; val_acc: 0.286000\n",
      "(Epoch 4 / 25) train acc: 0.578000; val_acc: 0.301000\n",
      "(Epoch 5 / 25) train acc: 0.690000; val_acc: 0.310000\n",
      "(Epoch 6 / 25) train acc: 0.656000; val_acc: 0.296000\n",
      "(Epoch 7 / 25) train acc: 0.686000; val_acc: 0.311000\n",
      "(Epoch 8 / 25) train acc: 0.748000; val_acc: 0.324000\n",
      "(Epoch 9 / 25) train acc: 0.744000; val_acc: 0.308000\n",
      "(Epoch 10 / 25) train acc: 0.786000; val_acc: 0.320000\n",
      "(Epoch 11 / 25) train acc: 0.764000; val_acc: 0.312000\n",
      "(Epoch 12 / 25) train acc: 0.800000; val_acc: 0.288000\n",
      "(Epoch 13 / 25) train acc: 0.828000; val_acc: 0.315000\n",
      "(Epoch 14 / 25) train acc: 0.806000; val_acc: 0.322000\n",
      "(Epoch 15 / 25) train acc: 0.852000; val_acc: 0.344000\n",
      "(Epoch 16 / 25) train acc: 0.868000; val_acc: 0.327000\n",
      "(Epoch 17 / 25) train acc: 0.892000; val_acc: 0.322000\n",
      "(Epoch 18 / 25) train acc: 0.876000; val_acc: 0.331000\n",
      "(Epoch 19 / 25) train acc: 0.890000; val_acc: 0.316000\n",
      "(Epoch 20 / 25) train acc: 0.916000; val_acc: 0.310000\n",
      "(Iteration 101 / 125) loss: 1.131315\n",
      "(Epoch 21 / 25) train acc: 0.918000; val_acc: 0.318000\n",
      "(Epoch 22 / 25) train acc: 0.908000; val_acc: 0.310000\n",
      "(Epoch 23 / 25) train acc: 0.922000; val_acc: 0.315000\n",
      "(Epoch 24 / 25) train acc: 0.912000; val_acc: 0.298000\n",
      "(Epoch 25 / 25) train acc: 0.920000; val_acc: 0.318000\n"
     ]
    }
   ],
   "source": [
    "# Train two identical nets, one with dropout and one without\n",
    "np.random.seed(231)\n",
    "num_train = 500\n",
    "small_data = {\n",
    "  'X_train': data['X_train'][:num_train],\n",
    "  'y_train': data['y_train'][:num_train],\n",
    "  'X_val': data['X_val'],\n",
    "  'y_val': data['y_val'],\n",
    "}\n",
    "\n",
    "solvers = {}\n",
    "dropout_choices = [0, 0.75]\n",
    "for dropout in dropout_choices:\n",
    "  model = FullyConnectedNet([500], dropout=dropout)\n",
    "  print(dropout)\n",
    "\n",
    "  solver = Solver(model, small_data,\n",
    "                  num_epochs=25, batch_size=100,\n",
    "                  update_rule='adam',\n",
    "                  optim_config={\n",
    "                    'learning_rate': 5e-4,\n",
    "                  },\n",
    "                  verbose=True, print_every=100)\n",
    "  solver.train()\n",
    "  solvers[dropout] = solver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4IAAAJNCAYAAABkwMu0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzs3X2cnHV56P/PxSbICpgFkqJ5QIJgKkIkniWoVOShGiwQotIYrCg+oT0irT2mQF8ac3JqTYlHbAo/z8En0CohUhpBaVMbRA/4AInBRMBIeDK7UE0CiWIXSML1+2Nmw+6yWXazO3PvzP15v155zd7f+557rpm5dzLXfr/f6xuZiSRJkiSpPPYpOgBJkiRJUn2ZCEqSJElSyZgISpIkSVLJmAhKkiRJUsmYCEqSJElSyZgISpIkSVLJmAhKkkohIloi4omIOKzoWCRJKlq4jqAkaTSKiCd6bL4QeArYVd3+YGZ+vf5RSZLUHEwEJUmjXkQ8BLw/M/9jgGPGZObO+kVVP8383CRJxXBoqCSpIUXE30bEdRFxbUT8DnhnRLw2In4cEdsi4tGIWBoRY6vHj4mIjIjDq9v/VN3/rxHxu4j4UURM3cNj7RMR10fEf1bPfWtEvKLH/hdGxOUR8auI2B4RP4iIF1T3nVSNaXtEbIqI86rtt0XE+T3O8f6IuLVPrP89IjYCv6i2XxERHRHx24i4MyJe1+P+YyLiExFxf3X/6oiYGBH/NyL+vs/zuTkiPjL8d0GS1KhMBCVJjewtwDeAccB1wE7gL4DxwInA6cAHB7j/O4BPAAcDvwL+1wDHfhs4Cngx8HPgaz32XQ5MB06onutvgGeqieXNwGeBQ4AZwPohPL/ZwPHAsdXtn1Qf52DgeuCb3QknMB84h8pzbgPeDzwJXAOcGxEBEBGHAicD1w4hDklSkzERlCQ1stsy86bMfCYzuzLzzsz8SWbuzMwHgKuANwxw/+szc3Vm7gC+DhzX30HV81+dmb/LzCeBhcB/i4j9I6IFOB+4KDMfzcxdmXlb9ZzvBP41M5dXY9qSmXcN4fn9XWY+npld1Ti+lpmPVYeJXga8CDiyeuz7gb/JzPuq8d5VPfaHVBLCk6vHnQv8R2ZuGUIckqQmYyIoSWpkm3puRMQfRsR3qkM4fwssotI7uCf/2ePn/wIO6O+gasXRyyLigep5N1Z3jQcOBfYF7u/nrlP20D5YfZ/fX0fELyJiO/A4sD/PPr+BHuurVJJSqrdf28NxkqSSMBGUJDWyvhXP/i+VYZtHZuaLgAVAjMDjvAv4E+BUKsNQu3vhAvg18DTwsn7ut2kP7QC/p1INtduL+zlm9/OLiFOAvwLeRmXo50HAEzz7/AZ6rK8Bb4mIGdVjbtrDcZKkkjARlCQ1kwOB7cDvq8VcBpofONTzPgVspZK8fap7R2buAq4GPhcRL672Hp5YLVLzT8DpEfG2ajGX8RHxqupd7wLeFhGtEfFy4L2DiGEnsAUYS2V46v499n8R+NuIeFlUHBcRB1djfBj4GZX5gt+sDm+VJJWYiaAkqZn8D+DdwO+o9A5eN0Ln/QrwSPXf3cAP++z/KHAvsAZ4DPg7Kks0PQicBVxcbf8pzxZ++QyVHr/fAF+mkjQO5GbgP4D7gIeA3wKP9ti/BFgBrKruuwrYr8f+a6qP7bBQSZLrCEqSVAYRcSrwJeCI9D9/SSo9ewQlSWpyEbEvlWU1vmASKEkCE0FJkppaRBxLpcLowcDSgsORJI0SDg2VJEmSpJKxR1CSJEmSSsZEUJIkSZJKZkzRAYyU8ePH5+GHH150GJIkSZJUiDVr1mzJzAmDObZpEsHDDz+c1atXFx2GJEmSJBUiIh4e7LEODZUkSZKkkjERlCRJkqSSMRGUJEmSpJIxEZQkSZKkkjERlCRJkqSSKSQRjIgvR8RvIuLne9gfEbE0IjZGxLqIeHW9Y5QkSZKkZlXU8hFXA1cAX93D/jcDR1X/nQB8vnorSZLUcFas7WTJyg08sq2LiW2tzJ81jTkzJhUdlqQSK6RHMDN/ADw2wCFnA1/Nih8DbRHxkvpEJ0mSNHJWrO3k0hvW07mtiwQ6t3Vx6Q3rWbG2s+jQJJXYaJ0jOAnY1GO7o9omSZIa3Iq1nZy4+BamXvIdTlx8S9MnREtWbqBrx65ebV07drFk5YaCIpKk4oaGjoiIuAC4AOCwww4rOBpJkvR8unvHuhOj7t4xoGmHSj6yrWtI7Roeh+FKgzNaewQ7gSk9tidX23rJzKsysz0z2ydMmFC34CRJ0t4pY+/YxLbWIbVr7zkMVxq80ZoI3gi8q1o99DXA9sx8tOigJEnS8BTZO1bUkNT5s6bROralV1vr2Bbmz5pWl8cvkzL+oUHaW4UMDY2Ia4GTgfER0QF8EhgLkJn/B7gZ+BNgI/BfwHuKiFOSJI2siW2tdPaT9NW6d6zIIand53e4Yu05DFcavEISwcw893n2J/DhOoUjSZLqZP6sab0SMqhP79hAPUX1SMjmzJhUSOJXtvlyRf2hoczKdo010/Nt6GIxkiQ1umb6UjEYRfWOlbGnqIyFeYr6Q0PRivocKds11mzP10RQkiSK+SLVbF8qBquI3rEy9hQV3QtahKKH4Zbtc6TIa6yI17rZfqdMBCVJpVfUF6lm+1IxmpWxp6iMvaBQ7DDcsn2OFHWNFfVaN9vv1GitGipJUt0UVWmw6C8VZVrYfc6MSXz6rccyqa2VACa1tfLptx7b1Am3y1bUVxk/R4q6xop6rZvtd8pEUJJUekV9kSryS0UZ11ubM2MSt19yKg8uPoPbLzm1qZNAcNmKeivj50hR11hRr3Wz/U6ZCEqSSq+oL1JFfqlwvbXmV8Ze0CKV8XOkqGusqNe62X6nnCMoSSq9ouaPFVnYouhhqaqPoubLlVEZP0e6H7/e11iRc36b6XfKRFCSVHpFfpEq6ktFGatoSrVUxs+RohSd/DaLqKzd3vja29tz9erVRYchSVJD6Ft1Dyp/UW/kYU6SVHYRsSYz2wdzrD2CkqRRo2yLqxfJv6hLUrmZCEqSRoWyLq5epLINJ5MkPcuqoZKkUcEqlpIk1Y+JoCRpVLCKpSRJ9ePQUEnSqGAVS6m5OOdXGt3sEZQkjQpFLoosaWR1z/nt3NZF8uyc3xVrO4sOTVKViaAkaVSYM2MSn37rsUxqayWASW2tLmUgNSjn/JbIuuVw+TGwsK1yu2550RFpkBwaKkkaNaxiKTWH0s75XbccVi2C7R0wbjKctgCmzy06qtpZtxxuugh2VN/X7Zsq29C8z7uJ3mN7BCVJkjSi9jS3ty5zfovqoepOirZvAvLZpKiZe8hWLXo2Cey2o6vS3oya7D02EZQkSdKIKmzOb5Ff1MuWFEGlV2wo7SOpiIS/yd5jh4ZKkp7Dan+ShqP786LunyMDfVGv9fC9opOiIoYrjptcTbr7aa+looakFvke10AhiWBEnA78A9ACfDEzF/fZ/1Lgy8AE4DHgnZnZmK+wJDWY7mp/3YUeuqv9ASaDkgatkDm/RX5RL1tSBJWEs+djA4xtrbTXUlEJf1HvcY3UfWhoRLQAVwJvBo4Gzo2Io/sc9hngq5k5HVgEfLq+UUpSeVntT1LD2tMX8np8UT9tQSUJ6qnopKjWps+Fs5bCuClAVG7PWtq8va9Fvcc1UkSP4ExgY2Y+ABARy4CzgXt6HHM08FfVn78HrKhrhJJUYqWt9iep8RXVQwXPJj/1HqJZ9HDF6XPrXzWzqJ65ot7jGikiEZwE9HznOoAT+hzzM+CtVIaPvgU4MCIOycyt9QlRksprYlsrnf0kfXWp9idJw1H0F/UyJUVFKjrhb9DEr6/RWizmY8AVEXE+8AOgE9jV96CIuAC4AOCwww6rZ3ySVHNFFWyZP2tarzmCUKdqf1Iza6K1xwatqOfcRF/UB6XIpKgoRSf8TSIys74PGPFaYGFmzqpuXwqQmf3OA4yIA4BfZOaAf9Zob2/P1atXj3S4klSIvgVboJKMffqtx9YlGbRqqDSC+hbzgMoX9XrMpSpKGZ9zkcr4hwb1KyLWZGb7oI4tIBEcA/wSOI1KT9+dwDsy8+4ex4wHHsvMZyLiU8CuzBzwzxomgpKayYmLb+l3eOaktlZuv+TUAiKStNcuP2YPQ/emwEd/Xv946qGMz1kaBYaSCNa9amhm7gQuBFYC9wLLM/PuiFgUEbOrh50MbIiIXwKHAp+qd5ySVCQLtkhNpOhiHkUo43OWGkwhcwQz82bg5j5tC3r8fD1wfb3jkqTRwoItUhMpYzGPMj5nqcHUvUdQkvT85s+aRuvYll5tFmyRGlSRa4+tW14ZprmwrXK7bnntHxOabr01qRmN1qqhklRq3YVZLNgiNYGiKhz2LdiyfVNlu2dMtWJVR2nUq3uxmFqxWIwkSRpQ2SorWrBFKp2hFIuxR1BSw3BJA0l7rcjesaJYsEXSAEwEJTWEvuvqdW7r4tIb1gPUPBk0AZWawKpFvde0g8r2qkXNmwhasEXSACwWI6khLFm5odfi6gBdO3axZOWGmj5udwLaua2L5NkEdMXazpo+rqQRVsbeMQu2SBqAiaCkhlDUunpFJaCSRtieesGauXds+lw4a2llTiBRuT1rafP2gEoaEoeGSmoIRa2r58LuUg0UUbTltAW95whCOXrHps818ZPUL3sEJTWEotbV21Oi6cLu0l7qLtqyfROQzxZtqfX6dvaOSVIv9ghKaghFras3f9a0XkVqwIXdpWEpsmiLvWOStJuJoKSGMWfGpLpX63Rhd2mElbFoiySNQiaCkvQ8ikhApablkgaSNCo4R1CSJNWPSxpI0qhgIihJkurHoi2SNCo4NFSSpCIVsZRC0SzaIkmFMxGUJKko3UspdFfR7F5KAUyUJEk15dBQSZKKMtBSCpIk1ZCJoCRJRXEpBUlSQUwEJUkqyp6WTHApBUlSjZkISpJUFJdSkCQVxERQkqSiFL2UwrrlcPkxsLCtcrtueX0eV5JUuEKqhkbE6cA/AC3AFzNzcZ/9hwHXAG3VYy7JzJvrHqgkSbVW1FIKViyVpFKre49gRLQAVwJvBo4Gzo2Io/sc9nFgeWbOAOYB/199o5QklU7ZesesWCpJpVbE0NCZwMbMfCAznwaWAWf3OSaBF1V/Hgc8Usf4JEll0907tn0TkM/2jjVzMmjFUkkqtSISwUnAph7bHdW2nhYC74yIDuBm4CP1CU2SVEpl7B2zYqkkldpoLRZzLnB1Zk4G/gT4WkQ8J9aIuCAiVkfE6s2bN9c9SElSkyhj75gVSyWp1IpIBDuBKT22J1fbenofsBwgM38E7AeM73uizLwqM9szs33ChAk1CleSSqhs8+XK2DtWdMVSSVKhiqgaeidwVERMpZIAzgPe0eeYXwGnAVdHxCuoJIJ2+UmjwIq1nSxZuYFHtnUxsa2V+bOmMWdG39HdamhlrCZ52oLezxnK0TtWVMVSSVLh6t4jmJk7gQuBlcC9VKqD3h0RiyJidvWw/wF8ICJ+BlwLnJ+ZWe9YJfW2Ym0nl96wns5tXSTQua2LS29Yz4q1fTv11dDKOF/O3jFJUslEs+RX7e3tuXr16qLDkJraiYtvoXNb13PaJ7W1cvslpxYQkWpiYRuV4s19BSzcVu9oJEnSIEXEmsxsH8yxo7VYjKRR6JF+ksCB2tWgyjhfTpKkkjERlDRoE9tah9SuBmU1SUmSmp6JoKRBmz9rGq1jW3q1tY5tYf6saQVFpJoocr5c2aqVSpJUkCKqhkpqUN3VQa0aWgJFVJMsY7VSSZIKYrEYqUG5jIOazuXHVJK/vsZNgY/+vP7xSJLUYIZSLMYeQakBdS/j0LVjF/DsMg6AyaAa1/aOobVLkqS95hxBqQEtWblhdxLYrWvHLpas3FBQRNIIsFqpJEl1YyIoNSCXcagzC5jUh9VKJUmqGxNBqQG5jEMddRcw2b4JyGcLmNQjGSxbAlpktVJJkkrGOYJSA5o/a1qvOYLgMg41s2rRs1Usu+3oqrTXMkEpawXNIqqVSpJUQvYISg1ozoxJfPqtxzKprZUAJrW18um3HmuhmFooqoDJQAmoJEnSMNkjKDWoOTMmmfjVw7jJe1jSoMYFTKygKUmSasgeQUkaSFEFTKygKUmSashEUJIGUlQBEytoSpKkGnJoqCQ9nyIKmHQ/3qpFleGg4yZXkkALqUiSpBFgIihJo5UVNCVJUo04NFSSJEmSSsZEUJIkSZJKxkRQkiRJkkrGRFCSJEmSSqaQRDAiTo+IDRGxMSIu6Wf/5RFxV/XfLyNiWxFxSpIkSVIzqnvV0IhoAa4E3gh0AHdGxI2ZeU/3MZn50R7HfwSYUe84JUmSJKlZFdEjOBPYmJkPZObTwDLg7AGOPxe4ti6RSZIkSVIJFJEITgI29djuqLY9R0S8FJgK3FKHuCRJkiSpFIaVCEbERyLioJEKph/zgOszc9ceHv+CiFgdEas3b95cwzCkUWjdcrj8GFjYVrldt7zoiCRJktQghtsjeCiVOX7LqwVgYhD36QSm9NieXG3rzzwGGBaamVdlZntmtk+YMGHQQUsNb91yuOki2L4JyMrtTRc1fzJo8itJkjQihpUIZubHgaOALwHnA/dFxN9FxMsGuNudwFERMTUi9qWS7N3Y96CI+EPgIOBHw4lRqqUVazs5cfEtTL3kO5y4+BZWrN3T3zRG2KpFsKOrd9uOrkp7sypr8itJklQDw54jmJkJ/Gf1304qydv1EXHZHo7fCVwIrATuBZZn5t0RsSgiZvc4dB6wrHp+adRZsbaTS29YT+e2LhLo3NbFpTesr08yuL1jaO3NoIzJryRJUo0Ma/mIiPgL4F3AFuCLwPzM3BER+wD3AX/d3/0y82bg5j5tC/psLxxObFKtLVm5ga4dvaevdu3YxZKVG5gzo9/6RyNn3ORqz1g/7c2qjMmvJElSjQy3R/Bg4K2ZOSszv5mZOwAy8xngzGFHJ41ij2zrGlL7iDptAYxt7d02trXS3qz2lOQ2c/IrSZJUI8NNBP8VeKx7IyJeFBEnAGTmvcM8tzSqTWxrHVL7iJo+F85aCuOmAFG5PWtppb1ZlTH5lSRJqpFhDQ0FPg+8usf2E/20SU1p/qxpXHrD+l7DQ1vHtjB/1rT6BDB9bnMnfn11P9dViyrDQcdNriSBZXoNJEmSRshwE8HoWcwlM5+JiOGeU2oI3fMAl6zcwCPbupjY1sr8WdNqPz+wzMqW/EqSJNXIcJO2ByLiIiq9gAD/HXhgmOeUGsacGZNM/CRJktRwhjtH8EPA66gsCN8BnABcMNygJI1iLuouSZLU8IbVI5iZv6Gy3p+kMuhe1L17Pb/uRd3BIZuSJEkNZLjrCO4HvA94JbBfd3tmvneYcUkajQZa1N1EUJIkqWEMd2jo14AXA7OA7wOTgd8NNyhJo5SLukuSJDWF4SaCR2bmJ4DfZ+Y1wBlU5glKakYu6i5JktQUhpsI7qjebouIY4BxwB8M85ySRisXdZckSWoKw10+4qqIOAj4OHAjcADwiWFHJWl0clF3SZKkprDXiWBE7AP8NjMfB34AHDFiUUkavVzUXZIkqeHt9dDQzHwG+OsRjEWSJEmSVAfDnSP4HxHxsYiYEhEHd/8bkcgkSZIkSTUx3DmCb6/efrhHW+IwUUmSJEkatYaVCGbm1JEKRJIkSZJUH8NKBCPiXf21Z+ZXh3NeSZIkSVLtDHdo6PE9ft4POA34KWAiKEmSJEmj1HCHhn6k53ZEtAHLhhWRJEmSJKmmhls1tK/fA84blCRJkqRRbLhzBG+iUiUUKknl0cDyQdzvdOAfgBbgi5m5uJ9j5gILq+f/WWa+YzixSpIkSZIqhjtH8DM9ft4JPJyZHQPdISJagCuBNwIdwJ0RcWNm3tPjmKOAS4ETM/PxiPiDYcYpSZIkSaoabiL4K+DRzHwSICJaI+LwzHxogPvMBDZm5gPV+ywDzgbu6XHMB4ArM/NxgMz8zTDjlCRJkiRVDXeO4DeBZ3ps76q2DWQSsKnHdke1raeXAy+PiNsj4sfVoaSSJEmSpBEw3B7BMZn5dPdGZj4dEfsO85xQieso4GRgMvCDiDg2M7f1PCgiLgAuADjssMNG4GElSZIkqfkNt0dwc0TM7t6IiLOBLc9zn05gSo/tydW2njqAGzNzR2Y+CPySSmLYS2ZelZntmdk+YcKEvXoCkiRJklQ2w00EPwT8TUT8KiJ+BVwMfPB57nMncFRETK32Hs4DbuxzzAoqvYFExHgqQ0UfGGaskiRJkiSGv6D8/cBrIuKA6vYTg7jPzoi4EFhJZfmIL2fm3RGxCFidmTdW970pIu6hMu9wfmZuHU6sUk2sWw6rFsH2Dhg3GU5bANPnFh2VJEmSNKDIzOc/ak93jvg74LLuuXsRcRDwPzLz4yMU36C1t7fn6tWr6/2wKrN1y+Gmi2BH17NtY1vhrKUmg5IkSaq7iFiTme2DOXa4Q0Pf3LOAS3W5hz8Z5jmlxrBqUe8kECrbqxYVE48kSZI0SMNNBFsi4gXdGxHRCrxggOOl5rG9Y2jtkiRJ0igx3OUjvg6sioivAAGcD1wz3KCkhjBuMmzf1H+7JEmSNIoNq0cwM/8e+FvgFcA0KkVeXjoCcUmj32kLKnMCexrbWmmXJEmSRrHhDg0F+DWQwJ8CpwL3jsA5pdFv+txKYZhxU4Co3FooRpIkSQ1gr4aGRsTLgXOr/7YA11GpQHrKCMYmjX7T55r4SZIkqeHs7RzBXwD/DzgzMzcCRMRHRywqaYhWrO1kycoNPLKti4ltrcyfNY05MyYVHZYkSZI0Ku3t0NC3Ao8C34uIL0TEaVSKxUh1t2JtJ5fesJ7ObV0k0Lmti0tvWM+KtZ1FhyZJkiSNSnuVCGbmisycB/wh8D3gL4E/iIjPR8SbRjJA6fksWbmBrh27erV17djFkpUbCopIkiRJGt2GWzX095n5jcw8C5gMrAUuHpHIpEF6ZFvXkNolSZKkshuJqqEAZObjmXlVZp42UueUBmNiW+uQ2iVJkqSyG7FEUCrK/FnTaB3b0qutdWwL82dNKygiSZIkaXTb26qh0qgxZ8YkJm36NlN+uoQ/yM38Jiaw6dXzOX7G6UWHJkmSJI1KJoJqfOuWc/z6TwJdEPBiNvPi9Z+Eww9yjT9JkiSpHw4NVeNbtQh29CkMs6Or0i5JkiTpOUwE1fi2dwytXZIkSSo5h4ZqxKxY28mSlRt4ZFsXE9tamT9rGnNmTKr9A4+bDNs39d8uSZIk6TnsEdSIWLG2k0tvWE/nti4S6NzWxaU3rGfF2s7aP/hpC2Bsn6UixrZW2iVJkiQ9h4mgRsSSlRvo2rGrV1vXjl0sWbmh9g8+fS6ctRTGTQGicnvWUgvFSJIkSXvg0FCNiEe2dQ2pfcRNn2viJ0mSJA1SIT2CEXF6RGyIiI0RcUk/+8+PiM0RcVf13/uLiFODN7GtdUjtkiRJkopT90QwIlqAK4E3A0cD50bE0f0cel1mHlf998W6Bqkhmz9rGq1jW3q1tY5tYf6saQVFJEmSJGlPihgaOhPYmJkPAETEMuBs4J4CYtEI6a4OWkjVUEmSJElDUkQiOAnoWeu/Azihn+PeFhEnAb8EPpqZ/awPoNFkzoxJJn6SJElSAxitVUNvAg7PzOnAd4Fr+jsoIi6IiNURsXrz5s11DVCSJEmSGlURiWAnMKXH9uRq226ZuTUzn6pufhH4b/2dKDOvysz2zGyfMGFCTYKVJEmSpGZTRCJ4J3BUREyNiH2BecCNPQ+IiJf02JwN3FvH+CRJkiSpqdV9jmBm7oyIC4GVQAvw5cy8OyIWAasz80bgooiYDewEHgPOr3ec2gvrlsOqRbC9A8ZNhtMWuLafJEmSNApFZhYdw4hob2/P1atXFx1Gea1bDjddBDt6LCA/thXOWmoyKEmSJNVBRKzJzPbBHDtai8Wo0axa1DsJhMr2qkXFxCNJkiRpj0wENTK2dwytXZIkSVJhTAQ1MsZNHlq7JEmSpMKYCGpknLagMiewp7GtlXZJkiRJo4qJoEbG9LmVwjDjpgBRubVQjCRJkjQq1X35CDWx6XNN/CRJkqQGYI+gJEmSJJWMiaAkSZIklYyJoCRJkiSVjImgJEmSJJWMiaAkSZIklYyJoCRJkiSVjImgJEmSJJWMiWAzWrccLj8GFrZVbtctLzoiSZIkSaOIC8o3m3XL4aaLYEdXZXv7pso2uNi7JEmSJMAeweazatGzSWC3HV2VdkmSJEnCRLD5bO8YWrskSZKk0jERbDbjJg+tXZIkSVLpmAg2m9MWsLNlv15NO1v2g9MWFBSQJEmSpNHGRLDJrNh1IpfseD8dz4znmQw6nhnPJTvez4pdJxYdmiRJkqRRopCqoRFxOvAPQAvwxcxcvIfj3gZcDxyfmavrGGLDWrJyA51Pv47reV2v9h+t3MCcGZMKikqSJEnSaFL3HsGIaAGuBN4MHA2cGxFH93PcgcBfAD+pb4SN7ZFtXUNqlyRJklQ+RQwNnQlszMwHMvNpYBlwdj/H/S/g74En6xlco5vY1jqkdkmSJEnlU0QiOAnY1GO7o9q2W0S8GpiSmd+pZ2DNYP6sabSObenV1jq2hfmzphUUkSRJkqTRppA5ggOJiH2AzwLnD+LYC4ALAA477LDaBtYguucBLlm5gUe2dTGxrZX5s6Y5P1CSJEnSbkUkgp3AlB7bk6tt3Q4EjgFujQiAFwM3RsTsvgVjMvMq4CqA9vb2rGXQjWTOjEkmfpIkSZL2qIihoXcCR0XE1IjYF5gH3Ni9MzO3Z+b4zDw8Mw8Hfgw8JwmUJEmSJO2duieCmbkTuBBYCdwLLM/MuyNiUUTMrnc8kiRJklQ2hcwRzMybgZv7tC3Yw7En1yMmSZIkSSqLIoaGSpIkSZIKZCIoSZIkSSVjIihJkiRJJWMiKEmSJEklYyIoSZIkSSVjIihJkiRJJWMiKEmSJEklYyIoSZIkSSVjIihJkiRJJWMiKEmSJEklYyIoSZIkSSVjIihJkiRJJWMiKEmSJEklYyIoSZIkSSVjIihJkiRJJWMiKEmSJEklYyIoSZIkSSVjIihJkiRJJWMiKEmSJEklYyIoSZIkSSVjIig2paWYAAAgAElEQVRJkiRJJVNIIhgRp0fEhojYGBGX9LP/QxGxPiLuiojbIuLoIuKUJEmSpGZU90QwIlqAK4E3A0cD5/aT6H0jM4/NzOOAy4DP1jlMSZIkSWpaRfQIzgQ2ZuYDmfk0sAw4u+cBmfnbHpv7A1nH+CRJkiSpqY0p4DEnAZt6bHcAJ/Q9KCI+DPwVsC9wan1CkyRJkqTmV0QiOCiZeSVwZUS8A/g48O6+x0TEBcAFAIcddlh9A5QkSdKQ7Nixg46ODp588smiQ5Ea2n777cfkyZMZO3bsXp+jiESwE5jSY3tytW1PlgGf729HZl4FXAXQ3t7u8FFJkqRRrKOjgwMPPJDDDz+ciCg6HKkhZSZbt26lo6ODqVOn7vV5ipgjeCdwVERMjYh9gXnAjT0PiIijemyeAdxXx/gkSZJUA08++SSHHHKISaA0DBHBIYccMuye9br3CGbmzoi4EFgJtABfzsy7I2IRsDozbwQujIg/BnYAj9PPsFBJkiQ1HpNAafhG4veokHUEM/PmzHx5Zr4sMz9VbVtQTQLJzL/IzFdm5nGZeUpm3l1EnMOxYm0nJy6+hamXfIcTF9/CirUDjX6VJElSPfzbv/0b06ZN48gjj2Tx4sX9HvPUU0/x9re/nSOPPJITTjiBhx56aPe+T3/60xx55JFMmzaNlStXPu/jXX311Vx44YUjFf6Qfe5zn+O//uu/Cnv8kTaY9++jH/0oxx13HMcddxwvf/nLaWtr272vpaVl977Zs2c/7+M99NBDHHPMMSMW/1CtWLGCe+65pybnHrXFYhrZirWdXHrDerp27AKgc1sXl96wHoA5MyYVGZokSVJp7dq1iw9/+MN897vfZfLkyRx//PHMnj2bo4/uvaT1l770JQ466CA2btzIsmXLuPjii7nuuuu45557WLZsGXfffTePPPIIf/zHf8wvf/lLWlpahhzLzp07GTOm9l/FP/e5z/HOd76TF77whTV/rFob7Pt3+eWX7/75H//xH1m7du3u7dbWVu66665hx1Kv92/FihWceeaZz3mOI6GQHsFmt2Tlht1JYLeuHbtYsnJDQRFJkiQ1npEeYXXHHXdw5JFHcsQRR7Dvvvsyb948vvWtbz3nuG9961u8+92VmUnnnHMOq1atIjP51re+xbx583jBC17A1KlTOfLII7njjjuec/+vfOUrvPzlL2fmzJncfvvtu9vPP/98PvShD3HCCSfw13/91zz22GPMmTOH6dOn85rXvIZ169YBsHDhQs477zxe+9rXctRRR/GFL3wBqBQJmT9/PscccwzHHnss1113HQC33norZ5555u7HufDCC7n66qtZunQpjzzyCKeccgqnnHLKsF67vbJuOVx+DCxsq9yuWz6s0w32/evp2muv5dxzzx3S46xZs4ZXvepVvOpVr+LKK6/c3X711Vcze/ZsTj31VE477bQB34+TTjqJM844g2nTpvGhD32IZ555Znc8xx57LMcccwwXX3zx7nMfcMABu3++/vrrOf/88/nhD3/IjTfeyPz58znuuOO4//77h/Q8no89gjXwyLauIbVLkiSpt1qMsOrs7GTKlGeL10+ePJmf/OQnAx43ZswYxo0bx9atW+ns7OQ1r3lNr/t3dvZOTh999FE++clPsmbNGsaNG8cpp5zCjBkzdu/v6Ojghz/8IS0tLXzkIx9hxowZrFixgltuuYV3vetdu3ur1q1bx49//GN+//vfM2PGDM444wx+9KMfcdddd/Gzn/2MLVu2cPzxx3PSSSft8fledNFFfPazn+V73/se48eP36vXbK+tWw43XQQ7qt9/t2+qbANMn7tXpxzs+9ft4Ycf5sEHH+TUU59dkvzJJ5+kvb2dMWPGcMkllzBnzpzn3O8973kPV1xxBSeddBLz58/vte+nP/0p69at4+CDD+af//mf9/h+3HHHHdxzzz289KUv5fTTT+eGG27gda97HRdffDFr1qzhoIMO4k1vehMrVqzoNwaA173udcyePZszzzyTc845Z0iv1WDYI1gDE9tah9QuSZKk3hp1hNVPfvITTj75ZCZMmMC+++7L29/+9l77//RP/3T3UNLbbruN8847D4BTTz2VrVu38tvf/haAs88+m9bWVsaPH88pp5zCHXfcwW233ca5555LS0sLhx56KG94wxu488476/sEB2vVomeTwG47uirtdbJs2TLOOeecXkN3H374YVavXs03vvEN/vIv//I5vWzbtm1j27ZtuxO67ven2xvf+EYOPvhggAHfj5kzZ3LEEUfQ0tLCueeey2233cadd965+9oYM2YMf/Znf8YPfvCDWr4EAzIRrIH5s6Zxzr4/5LZ9L+KBF7yD2/a9iHP2/SHzZ00rOjRJkqSGUIsRVpMmTWLTpk27tzs6Opg06bm9iz2P27lzJ9u3b+eQQw4Z9P0Hsv/++w/quL5VIQeqEjlmzJjdQw+BYS8rMCK2dwytfRCG+vovW7bsOcNCu48/4ogjOPnkk3vNHxyMWrx/fffX6/0zEayBOS23s3jsF5m8zxb2CZi8zxYWj/0ic1puf/47S5IkqSYjrI4//njuu+8+HnzwQZ5++mmWLVvWb+XI2bNnc8011wCV+VqnnnoqEcHs2bNZtmwZTz31FA8++CD33XcfM2fO7HXfE044ge9///ts3bqVHTt28M1vfnOP8bz+9a/n61//OlCZVzZ+/Hhe9KIXAZV5ik8++SRbt27l1ltv5fjjj+f1r3891113Hbt27WLz5s384Ac/YObMmbz0pS/lnnvu4amnnmLbtm2sWrVq92MceOCB/O53v9vr12yvjZs8tPZBGOz7B/CLX/yCxx9/nNe+9rW72x5//HGeeuopALZs2cLtt9/+nCIsbW1ttLW1cdtttwHsfn/6s6f3AypDQx988EGeeeYZrrvuOv7oj/6ImTNn8v3vf58tW7awa9curr32Wt7whjcAcOihh3LvvffyzDPP8C//8i+7H6OW759zBGth1SLG7OqdyY/Z9WSlK3wvx0RLkiSVyfxZ03rNEQRoHdsyrBFWY8aM4YorrmDWrFns2rWL9773vbzyla8EYMGCBbS3tzN79mze9773cd5553HkkUdy8MEHs2zZMgBe+cpXMnfuXI4++mjGjBnDlVde+ZyKoS95yUtYuHAhr33ta2lra+O4447bYzwLFy7kve99L9OnT+eFL3zh7uQTYPr06Zxyyils2bKFT3ziE0ycOJG3vOUt/OhHP+JVr3oVEcFll13Gi1/8YgDmzp3LMcccw9SpU3vNSbzgggs4/fTTmThxIt/73vf2+rUbstMW9J4jCDC2tdK+lwb7/kGlN3DevHm9etruvfdePvjBD7LPPvvwzDPPcMkll/RbjfMrX/kK733ve4kI3vSmN+0xnj29H7/4xS84/vjjufDCC9m4cSOnnHIKb3nLW9hnn31YvHgxp5xyCpnJGWecwdlnnw3A4sWLOfPMM5kwYQLt7e088cQTAMybN48PfOADLF26lOuvv56Xvexle/369RWZOWInK1J7e3uuXr266DAqFrYB/b2uAQu31TsaSZKkUeHee+/lFa94xaCPX7G2kyUrN/DIti4mtrUyf9a0UizFtXDhQg444AA+9rGPFR3K8KxbXukI2d5R6Qk8bUEpOkVuvfVWPvOZz/Dtb3+7po/T3+9TRKzJzPbB3N8ewVoYN7lSGam/dkmSJA3KnBmTSpH4Na3pc0uR+DUqE8FaqEFXuCRJksph4cKFRYegYTj55JM5+eSTiw7jeVksphamz4WzlsK4KUBUbs9a6l9EJEmSJI0K9gjWil3hkiRJz5GZz1tKX9LARqLOiz2CkiRJqov99tuPrVu3jsiXWKmsMpOtW7ey3377Des89ghKkiSpLiZPnkxHRwebN28uOhSpoe23335Mnjy8QpQmgpIkSaqLsWPHMnXq1KLDkIRDQyVJkiSpdEwEJUmSJKlkTAQlSZIkqWSiWao2RcRm4OGi4+jHeGBL0UGoqXmNqZa8vlRLXl+qJa8v1dJovb5empkTBnNg0ySCo1VErM7M9qLjUPPyGlMteX2plry+VEteX6qlZri+HBoqSZIkSSVjIihJkiRJJWMiWHtXFR2Amp7XmGrJ60u15PWlWvL6Ui01/PXlHEFJkiRJKhl7BCVJkiSpZEwEaygiTo+IDRGxMSIuKToeNZeIeCgi1kfEXRGxuuh41Pgi4ssR8ZuI+HmPtoMj4rsRcV/19qAiY1Tj2sP1tTAiOqufY3dFxJ8UGaMaV0RMiYjvRcQ9EXF3RPxFtd3PMA3bANdXQ3+GOTS0RiKiBfgl8EagA7gTODcz7yk0MDWNiHgIaM/M0biGjRpQRJwEPAF8NTOPqbZdBjyWmYurf9A6KDMvLjJONaY9XF8LgScy8zNFxqbGFxEvAV6SmT+NiAOBNcAc4Hz8DNMwDXB9zaWBP8PsEaydmcDGzHwgM58GlgFnFxyTJO1RZv4AeKxP89nANdWfr6HyH580ZHu4vqQRkZmPZuZPqz//DrgXmISfYRoBA1xfDc1EsHYmAZt6bHfQBBeMRpUE/j0i1kTEBUUHo6Z1aGY+Wv35P4FDiwxGTenCiFhXHTrqsD0NW0QcDswAfoKfYRphfa4vaODPMBNBqXH9UWa+Gngz8OHqsCupZrIyl8D5BBpJnwdeBhwHPAr872LDUaOLiAOAfwb+MjN/23Ofn2Earn6ur4b+DDMRrJ1OYEqP7cnVNmlEZGZn9fY3wL9QGY4sjbRfV+dGdM+R+E3B8aiJZOavM3NXZj4DfAE/xzQMETGWypf0r2fmDdVmP8M0Ivq7vhr9M8xEsHbuBI6KiKkRsS8wD7ix4JjUJCJi/+pkZSJif+BNwM8Hvpe0V24E3l39+d3AtwqMRU2m+wt61Vvwc0x7KSIC+BJwb2Z+tscuP8M0bHu6vhr9M8yqoTVULSH7OaAF+HJmfqrgkNQkIuIIKr2AAGOAb3h9abgi4lrgZGA88Gvgk8AKYDlwGPAwMDczLfihIdvD9XUylSFVCTwEfLDHfC5p0CLij4D/B6wHnqk2/w2VeVx+hmlYBri+zqWBP8NMBCVJkiSpZBwaKkmSJEklYyIoSZIkSSVjIihJkiRJJWMiKEmSJEklYyIoSZIkSSVjIihJUh8RsSsi7urx75IRPPfhEdFQa01JkprPmKIDkCRpFOrKzOOKDkKSpFqxR1CSpEGKiIci4rKIWB8Rd0TEkdX2wyPilohYFxGrIuKwavuhEfEvEfGz6r/XVU/VEhFfiIi7I+LfI6K1sCclSSolE0FJkp6rtc/Q0Lf32Lc9M48FrgA+V237R+CazJwOfB1YWm1fCnw/M18FvBq4u9p+FHBlZr4S2Aa8rcbPR5KkXiIzi45BkqRRJSKeyMwD+ml/CDg1Mx+IiLHAf2bmIRGxBXhJZu6otj+ameMjYjMwOTOf6nGOw4HvZuZR1e2LgbGZ+be1f2aSJFXYIyhJ0tDkHn4eiqd6/LwL5+xLkurMRFCSpKF5e4/bH1V//iEwr/rznwH/r/rzKuDPASKiJSLG1StISZIG4l8gJUl6rtaIuKvH9r9lZvcSEgdFxDoqvXrnVts+AnwlIuYDm4H3VNv/ArgqIt5Hpefvz4FHax69JEnPwzmCkiQNUnWOYHtmbik6FkmShsOhoZIkSZJUMvYISpIkSVLJ2CMoSSqV6uLvGRHOk5cklZaJoCSpoUTEv0XEon7az46I/zTBkyTp+ZkISpIazTXAOyMi+rSfB3w9M3cWENOIiAr/b5Yk1Zz/2UiSGs0K4BDg9d0NEXEQcCbw1er2GRGxNiJ+GxGbImLhYE8eEZdExP0R8buIuCci3tJn/wci4t4e+19dbZ8SETdExOaI2BoRV1TbF0bEP/W4f6+hqRFxa0R8KiJuB/4LOCIi3tPjMR6IiA/2ieHsiLir+vzuj4jTI+JPI2JNn+P+KiK+NdjnLkkqDxNBSVJDycwuYDnwrh7Nc4FfZObPqtu/r+5vA84A/jwi5gzyIe6nkmSOA/4n8E8R8RKAiPhTYGH13C8CZgNbI6IF+DbwMHA4MAlYNoSndR5wAXBg9Ry/oZLYvojKmoSX90g4Z1JJeOdXn99JwEPAjcDUiHhFn/N+dQhxSJJKwkRQktSIrgHOiYj9qtvvqrYBkJm3Zub6zHwmM9cB1wJvGMyJM/ObmflI9b7XAfcBM6u73w9clpl3ZsXGzHy4un8iMD8zf5+ZT2bmbUN4Pldn5t2ZuTMzd2TmdzLz/upjfB/4d57tAX0f8OXM/G41xs7M/EVmPgVcB7wTICJeSSUp/fYQ4pAklYSJoCSp4VSTrC3AnIh4GZVE7Bvd+yPihIj4XnWY5nbgQ8D4wZw7It5VHXa5LSK2Acf0uO8UKj2GfU0BHh7G/MRNfWJ4c0T8OCIeq8bwJ4OIASrJ8Duq8yfPA5ZXE0RJknoxEZQkNaqvUukJfCewMjN/3WPfN6gMlZySmeOA/wP0LS7zHBHxUuALwIXAIZnZBvy8x303AS/r566bgMP2ULH098ALe2y/uJ9jdi/qGxEvAP4Z+AxwaDWGmwcRA5n5Y+BpKr2H7wC+1t9xkiSZCEqSGtVXgT8GPkCPYaFVBwKPZeaT1Tl17xjkOfenkpRtBoiI91DpEez2ReBjEfHfqhU+j6wmj3cAjwKLI2L/iNgvIk6s3ucu4KSIOCwixgGXPk8M+wIvqMawMyLeDLypx/4vAe+JiNMiYp+ImBQRf9hj/1eBK4AdQxyeKkkqERNBSVJDysyHgB9SSd5u7LP7vwOLIuJ3wAIqxWUGc857gP8N/Aj4NXAscHuP/d8EPkWlx/F3VCqYHpyZu4CzgCOBXwEdwNur9/kulbl764A1PM+cvcz8HXBRNebHqSSxN/bYfwfVAjLAduD7wEt7nOJrVJLXf0KSpD2IzHz+oyRJUkOIiFYqVUdfnZn3FR2PJGl0skdQkqTm8ufAnSaBkqSB9DepXZIkNaCIeIhKUZnBrpkoSSoph4ZKkiRJUsk4NFSSJEmSSsZEUJIkSZJKpmnmCI4fPz4PP/zwosOQJEmSpEKsWbNmS2ZOGMyxTZMIHn744axevbroMCRJkiSpEBHx8GCPdWioJEmSJJWMiaAkSZIklYyJoCRJkiSVjImgJEmSJJWMiaAkSZIklYyJoCRJkiSVjImgJEmqr3XL4fJjYGFb5Xbd8qIjkqTSaZp1BCVJUgNYtxxuugh2dFW2t2+qbANMn1tcXJJUMvYISpKk+lm16NkksNuOrkq7JKluTAQlSVL9bO8YWrskqSZMBCVJUv2Mmzy0dklSTdQ0EYyI0yNiQ0RsjIhL+tn/oYhYHxF3RcRtEXF0n/2HRcQTEfGxWsYpSZLq5LQFMLa1d9vY1kq7JKluapYIRkQLcCXwZuBo4Ny+iR7wjcw8NjOPAy4DPttn/2eBf61VjJIkqc6mz4WzlsK4KUBUbs9aaqEYSaqzWlYNnQlszMwHACJiGXA2cE/3AZn52x7H7w9k90ZEzAEeBH5fwxglSVK9TZ9r4idJBavl0NBJwKYe2x3Vtl4i4sMRcT+VHsGLqm0HABcD/7OG8UmSJElSKRVeLCYzr8zMl1FJ/D5ebV4IXJ6ZTwx034i4ICJWR8TqzZs31zhSSZIkSWoOtRwa2glM6bE9udq2J8uAz1d/PgE4JyIuA9qAZyLiycy8oucdMvMq4CqA9vb2RJIkSZL0vGqZCN4JHBURU6kkgPOAd/Q8ICKOysz7qptnAPcBZObrexyzEHiibxIoSZIkSdo7NUsEM3NnRFwIrARagC9n5t0RsQhYnZk3AhdGxB8DO4DHgXfXKh5JkiRJUkVkNseIyvb29ly9enXRYUiSJElSISJiTWa2D+bYwovFSJIkSZLqy0RQkiRJkkrGRFCSJEmSSsZEUJIkSZJKxkRQkiSpWa1bDpcfAwvbKrfrlhcdkaRRopbrCEqSJKko65bDTRfBjq7K9vZNlW2A6XOLi0vSqGCPoCRJUjNatejZJLDbjq5Ku6TSMxGUJElqRts7htYuqVRMBCVJkprRuMlDa5dUKiaCkiRJzei0BTC2tXfb2NZKu6TSMxGUJEnlULYKmtPnwllLYdwUICq3Zy21UIwkwKqhkiSpDMpaQXP63OZ+fpL2mj2CkiSp+VlBU5J6MRGUJEnNzwqaktSLiaAkPZ+yzSuSmpEVNCWpFxNBSRpI97yi7ZuAfHZekcmg1FisoClJvVgsRpIGMtC8IgswSI2j+/d11aLKcNBxkytJoL/HagIr1nayZOUGHtnWxcS2VubPmsacGZOKDkujnImgJA3EeUVqZuuWlysxKrKCZtlea9XNirWdXHrDerp27AKgc1sXl96wHsBkUANyaKgkDcR5RWpWDnuuH19r1dCSlRt2J4HdunbsYsnKDQVFpEZR00QwIk6PiA0RsTEiLuln/4ciYn1E3BURt0XE0dX2N0bEmuq+NRFxai3jlKQ9cl6RmpXLKdSPr7Vq6JFtXUNql7rVLBGMiBbgSuDNwNHAud2JXg/fyMxjM/M44DLgs9X2LcBZmXks8G7ga7WKU5IGNH0unLUUxk0BonJ71tLmH9JlpdTm57Dn+vG1Vg1NbGsdUrvUrZZzBGcCGzPzAYCIWAacDdzTfUBm/rbH8fsDWW1f26P9bqA1Il6QmU/VMF5J6l+R84qK0D2MrbsHo3sYG5TrdWh24yZXhyr2066R5WutGpo/a1qvOYIArWNbmD9rWoFRqRHUcmjoJKDnp15Hta2XiPhwRNxPpUfwon7O8zbgpyaBklQnDmMrB4c914+vtWpozoxJfPqtxzKprZUAJrW18um3HmuhGD2vwquGZuaVwJUR8Q7g41SGggIQEa8E/h54U3/3jYgLgAsADjvssNoHK0ll4DC2cnA5hfrxtS6PgqrDzpkxycSvXpqoAnAtE8FOYEqP7cnVtj1ZBny+eyMiJgP/ArwrM+/v7w6ZeRVwFUB7e3sON2BJEg5jK5OyDXsukq9183NYffNrsve4lkND7wSOioipEbEvMA+4secBEXFUj80zgPuq7W3Ad4BLMvP2GsYoSerLYWySNHQOq29+TfYe1ywRzMydwIXASuBeYHlm3h0RiyJidvWwCyPi7oi4C/grnh0WeiFwJLCgurTEXRHxB7WKVZLUQ1krpf7/7d17lFxlmej/70MnMa0oDZJBkwAGA/mJJJA5TRBQIIAkHjBEjgNhvIDgoHNEdFxGwnHE/Dgzh/yIZ3QQ1izxBnOWGpDBJo6XDCeADghCYjAMwQhyke4wkkSCIk3I5fn9UdWhu8mlOund1VX7+1mrV9d+a++qp7p2duqp93nfV9LgKtvsw5bVN78me48LHSOYmT8Eftiv7fJetz+xg+P+Dvi7ImOTJO2EZWxDp4nGm0jbNFkJXS1eaH0Dr+5+evvtdYhnyJTpGtZkQycKXVBekiTtRM+H5eeeAvLlD8vN3nOi5tdkJXS1uGrTObyQo/q0vZCjuGrTOXWKaAiU7RrWZEMnTAQlSaqXEn5YVkk0WQldLW54fhrzNn2Yzq37szWDzq37M2/Th7nh+Wn1Dq04ZbuGNdnQibovHyFJUmmV8MOySqLJSuhqMbatlcUb3s7il97ep31cW+sOjmgCZbyGNdHQCXsEJUnDR9kml9jRh+Im/rCskmiyErpazJ0xidaRLX3aWke2MHfGpDpFNAS8hjU0E0FJ0vBQtrEmUMoPyyqJJiuhq8XsqeO48qzJjGtrJaj0BF551uTmXujda1hDi8zmWIe9vb09ly1bVu8wJBWpTDOTldEXj9hBKdmB8Df/MfTxDBXPa0mNzGvYsBIRyzOzvZZ9HSMoNaqyXXhLOBV56ZRxrAk01XgT7VjHii4WLlnNmg3djG1rZe6MSc3dU1RPZfv/Eer7mr2GNSwTQakRlTEp2tnMZM36msumhJNLaGjVKxnrWNHFZbc8SPemLQB0bejmslseBDAZHGxl/P+xjK9Zg8IxglIjKtt0zVDe3qIycayJCtSTjHVt6CZ5ORnrWNFV+HMvXLJ6WxLYo3vTFhYuWV34c5dOGf9/LONr1qAwEZQaURmTojLOTFa2GTRLOLmEhk49k7E1G7oH1N4sOlZ0cfyC25kw7wccv+D2IUm6S/n/YxlfswaFpaFSIypjCd0pl/ctfYHm7i0qa6mPY01UkHomY2PbWunazvOMbeL15epWDlvG/x/L+Jo1KOwRlBpRGUvoytZbZKmPNKh2lHQNRTJWxvXl6tYDW8b/H8v4mjUo7BGUGlFP8lO2WdHK1FtkqU9pOJvk0Jg7Y1KfHioYumSs5/0s0/tctx7YMv7/WMbXrEFhIig1qjIlRWVkqc+Qq0dC5mySQ6feydjsqeNK9Z7Wsxy2Y8vxLNx4NWte7Gbs6FbmbpnE7MKftc78TKDdYCIoScNR2cZE1lm9ErKdlc+VKWkYKmVLxuqpXj2wfrki1c4xgpI0HJVtTGSd1Ws8U1lnk1Tzmz11HFeeNZlxba0EMK6tlSvPmlx4MuZSHVLt7BGUpOHKUp8hU6+ErIyzSao86tED65crUu3sEZQklV69ZpQs42ySUpHqOTus1GhMBCVJpVevhKxe5XNSs/LLFal2hZaGRsRM4B+BFuBrmbmg3/0fBT4GbAGeBy7KzFXV+y4DLqzed0lmLikyVmm3rLzJ6ZqlJlDPGSWdwEQaPPWeHVZqJJGZxTxwRAvwa+CdQCdwP3BuT6JX3ed1mfmH6u1ZwH/PzJkRcTjwHWAaMBb4v8BhmbmFHWhvb89ly5YV8lqk7Vp50/ZndXRCD0mSJNVBRCzPzPZa9i2yNHQa8GhmPpaZLwGLgDN779CTBFa9BujJSs8EFmXmxsx8HHi0+njS8LH0ir5JIFS2l15Rn3gkSZKkGhVZGjoO6L0acidwTP+dIuJjwKeAUcDJvY69t9+x9ulreHmuc2DtkiRJ0jBR98liMvs+TZIAACAASURBVPPazHwzcCnwtwM5NiIuiohlEbFs7dq1xQQo7cg+4wfWLkmSpMGx8ib44hEwv63ye+VN9Y6o4RSZCHYBB/baHl9t25FFwOyBHJuZ12Vme2a2jxkzZg/DlQbolMvZ3DK6T9PmltGVCWMkSZJKoGNFF8cvuJ0J837A8Qtup2PFzj7uD5KeeRqeewrIyu/vX2IyOEBFJoL3A4dGxISIGAXMARb33iEiDu21eTrwSPX2YmBORLwqIiYAhwL3FRirNGAdW45n3qYP07l1f7Zm0Ll1f+Zt+jAdW46vd2iSJEmF61jRxWW3PEjXhm4S6NrQzWW3PFh8Mug8DYOisDGCmbk5Ii4GllBZPuIbmflQRFwBLMvMxcDFEXEqsAl4FjiveuxDEXETsArYDHxsZzOGSvWwcMlqul46jps5rk/7PUtWN/c01S6ZIUkNo2NFl0spqDALl6yme1Pfj+jdm7awsOjPQs7TMCgKXUcwM38I/LBf2+W9bn9iJ8f+PfD3xUUn7Zk1G7oH1N4U+i+Z0VOKASaDkjTM9PTW9HxQ7+mtAUwGNSjq9llon/HVstDttKtmdZ8sRmpUY9taB9TeFCzFkKSGsbPeGmkw1O2z0CmXV9Zu7m1kq/M0DJCJoLSb5s6YROvIlj5trSNbmDtjUp0iGgKWYkhSwyhl5YqGVN0+C005G959NexzIBCV3+++2uqkASq0NFRqZj1lNaUae2EphiQ1jLFtrXRtJ+lr6soVDam6fhaacraJ3x6KzKx3DIOivb09ly1bVu8wpCFTlwkA+o8RhEopht/CSdKw03+MIFR6a648a3Jzf2kplVhELM/M9lr2tUdQakB1mwCgJ9lz1lBJGvZKWbkiqWb2CEoN6PgFt2+33GdcWyt3zzu5DhFJkiSp3uwRlJqcEwBIamSubSepUTXT9ctEUGpATgAgqVG5tp2kRtVs1y+Xj5AaUCmXrpDUFFzbTtJg6FjRxfELbmfCvB9w/ILb6VjRVfhzNtv1yx5BqQE5AYBUgJU3ORHSELC0XdKeqlfPXLNdv0wEpQY1e+o4Ez9psPRfGuW5pyrbYDI4yCxtl7SndtYzV+Rno2a7flkaKknS0iv6ro8Jle2lV9QnniZmabukPVWvnrlmu37ZI6jmYEmXpD3xXOfA2rXbLG2XtKfq1TPXbNcvE0E1Pku6pKZRt2m59xlfuXZsr12DztJ2SXti7oxJfcYIwtD1zDXT9cvSUDU+S7qkptAz+L9rQzfJy4P/h2ImOE65HEb2+yZ5ZGulXZI0rMyeOo4rz5rMuLZWAhjX1sqVZ01umgRtqNgjqMFTr/JMS7qkplCvwf/Ay9cqS8wlqSE0U89cvZgIanDUszzTki4VrG7liiVT92m5p5xt4idJKg1LQzU46lmeaUmXClTXcsWS2dEg/0adlluSpOHMRFCDo57lmVPOhndfDfscCETl97uv9pt9DYqdlStqcDXbtNySJA1nhZaGRsRM4B+BFuBrmbmg3/2fAj4MbAbWAhdk5pPV+64CTqeSrN4GfCIzs8h4tQfqXZ5pSZcKUs9yxbKVpDbbtNySJA1nhSWCEdECXAu8E+gE7o+IxZm5qtduK4D2zHwhIv4auAo4JyKOA44HplT3uws4EbizqHi1h065vO8YQbA8U02hXmsV9ZSk9vRG9pSkAk2dGDn4f2iV7csGSdLLdlkaGhEfj4h9d+OxpwGPZuZjmfkSsAg4s/cOmXlHZr5Q3bwX6Ok+SmA0MAp4FTAS+N1uxKChYnmmmlS9yhUtSVXRHP8qSeVWS4/gAVR6834BfANYUmOJ5jigd61gJ3DMTva/EPgRQGbeExF3AE8DAVyTmQ/X8JyqJ8sz1YTqVa5Y9xk01fTqulyHJKnudpkIZubfRsTngNOADwHXRMRNwNcz8zeDEUREvB9op1L+SURMBN7Cyz2Et0XEOzLz3/sddxFwEcBBBx00GKFI0ivUo1yxXiWpKg+/bJCkcqtp1tBqD+B/Vn82A/sCN1cndNmRLuDAXtvjq219RMSpwGeBWZm5sdr8HuDezHw+M5+n0lN47Hbiui4z2zOzfcyYMbW8FElqCM6gqaK5XIcklVstYwQ/ERHLqUzkcjcwOTP/GvgvwH/byaH3A4dGxISIGAXMARb3e+ypwFeoJIHP9Lrrt8CJETEiIkZS6Sm0NFRSacyeOo4rz5rMuLZWAhjX1sqVZ022ZE+Dxi8bJKncahkjuB9wVs+yDj0yc2tEnLGjgzJzc0RcDCyhsnzENzLzoYi4AliWmYuBhcDewHcjAuC3mTkLuBk4GXiQysQxP87M7w/85UlS43IGTRXJ5TokqdxiV/O+RMTbgIcy84/V7dcBb8nMnw9BfDVrb2/PZcuW1TsMSZIkSaqLiFieme217FvLGMF/Ap7vtf18tU2SJEmS1IBqSQSj93IRmbmVAheilyRJkiQVq5ZE8LGIuCQiRlZ/PgE8VnRgkiRJkqRi1JIIfhQ4jsrSDz2Lwl9UZFCSJEmSpOLUsqD8M1SWfpAkSZIkNYFdJoIRMRq4EHgrMLqnPTMvKDAuSZIkSVJBaikN/T/AG4AZwE+A8cAfiwxKkiRJklScWhLBiZn5OeBPmXkDcDqVcYKSJEmSpAZUyzIQm6q/N0TEEcB/An9WXEhqVB0ruli4ZDVrNnQztq2VuTMmMXvquHqHJUmSJKmfWhLB6yJiX+BvgcXA3sDnCo1KDadjRReX3fIg3Zu2ANC1oZvLbnkQwGRQkiRJGmZ2WhoaEXsBf8jMZzPzp5l5SGb+WWZ+ZYjiU4NYuGT1tiSwR/emLSxcsrpOEUmSJEnakZ0mgpm5FfjMEMWiBrZmQ/eA2iVJkiTVTy2lof83Ij4N3Aj8qacxM39fWFRqOGPbWunaTtI3tq21DtGoSI4FLQffZ0mSmlstieA51d8f69WWwCGDH44a1dwZk/qMEQRoHdnC3BmT6hiVBptjQcvB91mSpOa3y+UjMnPCdn5MAtXH7KnjuPKsyYxrayWAcW2tXHnWZD80NhnHgpaD77MkSc1vlz2CEfHB7bVn5j8PfjhqZLOnjjPxa3KOBS0H32dJkppfLaWhR/e6PRo4BfgFYCIolYxjQcvB91mSpOZXS2nox3v9/BXw51TWEtQw1bGii+MX3M6EeT/g+AW307Giq94hqUnMnTGJ1pEtfdocC9p8fJ8lSWp+tfQI9vcnYMJgB6LB4SQPKlLPOeRsks3N91mSpOYXmbnzHSK+T2WWUKj0IB4O3JSZ8wqObUDa29tz2bJl9Q6j7o5fcPt2S7rGtbVy97yT6xCRJEmSpKEQEcszs72WfWvpEfxCr9ubgSczs7PGQGYC/wi0AF/LzAX97v8U8OHq464FLsjMJ6v3HQR8DTiQSiL6XzPziVqet8yc5EGSJEnSruxyjCDwW+DnmfmTzLwbWB8Rb9rVQRHRAlwLvItKL+K5EXF4v91WAO2ZOQW4Gbiq133/DCzMzLcA04Bnaoi19HY0mYOTPEiSJEnqUUsi+F1ga6/tLdW2XZkGPJqZj2XmS8Ai4MzeO2TmHZn5QnXzXmA8QDVhHJGZt1X3e77XftoJJ3mQJEmStCu1JIIjqokcANXbo2o4bhzwVK/tzmrbjlwI/Kh6+zBgQ0TcEhErImJhtYdRu+DC7pIkSZJ2pZYxgmsjYlZmLgaIiDOBdYMZRES8H2gHTuwV1zuAqVRKU28Ezge+3u+4i4CLAA466KDBDKmhubC7JEmSpJ2ppUfwo8D/iIjfRsRvgUuBj9RwXBeViV56jK+29RERpwKfBWZl5sZqcyfwQLWsdDPQQWX9wj4y87rMbM/M9jFjxtQQkiRJkiRplz2Cmfkb4G0RsXd1+/kaH/t+4NCImEAlAZwD/GXvHSJiKvAVYGZmPtPv2LaIGJOZa4GTAdeGkCRJkqRBsMsewYj4XxHRVp2w5fmI2Dci/m5Xx1V78i4GlgAPU1l78KGIuCIiZlV3WwjsDXw3Ih6IiMXVY7cAnwaWRsSDQABf3a1XKEmSJEnqo5YF5Vdk5tR+bb/IzFeUataTC8pLkiRJKrOBLChfyxjBloh4Va8HbwVetZP9JUmSJEnDWC2zhn6LSonmN6mUaJ4P3FBkUJIkSZKk4tQyWcz/FxG/BE4FksqYv4OLDkySJEmSVIxaSkMBfkclCfwLKjN4PlxYRNpzK2+CLx4B89sqv1feVO+IJEmSJA0jO+wRjIjDgHOrP+uoLOoemTl9iGLT7lh5E3z/EtjUXdl+7qnKNsCUs+sXlyRJkqRhY2c9gr+i0vt3Rma+PTO/DGwZmrC025Ze8XIS2GNTd6VdkiRJkth5IngW8DRwR0R8NSJOoTJZjIaz5zoH1i5JkiSpdHaYCGZmR2bOAf4f4A7gk8CfRcQ/RcRpQxWgBmif8QNrlyRJklQ6u5wsJjP/lJnfzsx3A+OBFcClhUem3XPK5TCytW/byNZKuyRJkiRR2zqC22Tms8B11R8NRz0Twiy9olIOus/4ShLY5BPFdKzoYuGS1azZ0M3YtlbmzpjE7Knj6h2WJEmSNCwNKBFUg5hydtMnfr11rOjislsepHtTZS6jrg3dXHbLgwAmg5IkSdJ21LqOoDRsLVyyelsS2KN70xYWLlldp4gkSZKk4c1EUA1vzYbuAbVLkiRJZWciqIY3tq11QO2SJElS2ZkIquHNnTGJ1pEtfdpaR7Ywd8akOkUkSZIkDW9OFqOG1zMhjLOGSpIkSbUxEVRTmD11nImfJEmSVCNLQyVJkiSpZEwEJUmSJKlkCk0EI2JmRKyOiEcjYt527v9URKyKiJURsTQiDu53/+siojMirikyTkmNoWNFF8cvuJ0J837A8Qtup2NFV71DkiRJakiFJYIR0QJcC7wLOBw4NyIO77fbCqA9M6cANwNX9bv/fwI/LSpGSY2jY0UXl93yIF0bukmga0M3l93yoMmgJEnSbiiyR3Aa8GhmPpaZLwGLgDN775CZd2TmC9XNe4HxPfdFxH8BDgD+rcAYJTWIhUtW071pS5+27k1bWLhkdZ0ikiRJalxFJoLjgKd6bXdW23bkQuBHABGxF/C/gU8XFp2khrJmQ/eA2iVJkrRjw2KymIh4P9AOLKw2/Xfgh5nZuYvjLoqIZRGxbO3atUWHKamOxra1DqhdkiRJO1ZkItgFHNhre3y1rY+IOBX4LDArMzdWm48FLo6IJ4AvAB+MiAX9j83M6zKzPTPbx4wZM9jxSxpG5s6YROvIlj5trSNbmDtjUp0ikiRJalxFLih/P3BoREygkgDOAf6y9w4RMRX4CjAzM5/pac/M9/Xa53wqE8q8YtZRSeUxe2qlsnzhktWs2dDN2LZW5s6YtK1dkiRJtSssEczMzRFxMbAEaAG+kZkPRcQVwLLMXEylFHRv4LsRAfDbzJxVVEySGtvsqeNM/CRJkgZBZGa9YxgU7e3tuWzZsnqHIUmSJEl1ERHLM7O9ln2HxWQxkiRJkqShYyIoSZIkSSVjIihJkiRJJWMiKEmSJEklYyIoSZIkSSVjIihJkiRJJWMiKEmSJEklYyIoSZIkSSVjIihJkiRJJWMiKEmSJEklYyIoSZIkSSVjIihJkiRJJWMiKEmSJEklYyIoSZIkSSVjIihJkiRJJWMiKEmSJEklYyIoSZIkSSVjIihJkiRJJWMiKEmSJEklU2giGBEzI2J1RDwaEfO2c/+nImJVRKyMiKURcXC1/aiIuCciHqred06RcUqSJElSmRSWCEZEC3At8C7gcODciDi8324rgPbMnALcDFxVbX8B+GBmvhWYCXwpItqKilWSJEmSyqTIHsFpwKOZ+VhmvgQsAs7svUNm3pGZL1Q37wXGV9t/nZmPVG+vAZ4BxhQYqyRJkiSVRpGJ4DjgqV7bndW2HbkQ+FH/xoiYBowCfjOo0UmSJElSSY2odwAAEfF+oB04sV/7G4H/A5yXmVu3c9xFwEUABx100BBEKkmSJEmNr8gewS7gwF7b46ttfUTEqcBngVmZubFX++uAHwCfzcx7t/cEmXldZrZnZvuYMVaOSpIkSVItikwE7wcOjYgJETEKmAMs7r1DREwFvkIlCXymV/so4HvAP2fmzQXGKEmSJEmlU1gimJmbgYuBJcDDwE2Z+VBEXBERs6q7LQT2Br4bEQ9ERE+ieDZwAnB+tf2BiDiqqFglSZIkqUwiM+sdw6Bob2/PZcuW1TsMSZIkSaqLiFieme217FvogvKSJEmSpOHHRFCSJEmSSsZEUJIkSZJKxkRQkiRJkkrGRFCSJEmSSsZEUJIkSZJKxkRQkiRJkkrGRFCSJEmSSsZEUJIkSZJKxkRQkiRJkkrGRFCSJEmSSsZEUJIkSZJKxkRQkiRJkkrGRFCSJEmSSsZEUJIkSZJKxkRQkiRJkkrGRFCSJEmSSsZEUJIkSZJKxkRQkiRJkkrGRFCSJEmSSqbQRDAiZkbE6oh4NCLmbef+T0XEqohYGRFLI+LgXvedFxGPVH/OKzJOSZIkSSqTwhLBiGgBrgXeBRwOnBsRh/fbbQXQnplTgJuBq6rH7gd8HjgGmAZ8PiL2LSpWSZIkSSqTInsEpwGPZuZjmfkSsAg4s/cOmXlHZr5Q3bwXGF+9PQO4LTN/n5nPArcBMwuMVZIkSZJKo8hEcBzwVK/tzmrbjlwI/Gg3j5UkSZIk1WhEvQMAiIj3A+3AiQM87iLgIoCDDjqogMgkSZIkqfkU2SPYBRzYa3t8ta2PiDgV+CwwKzM3DuTYzLwuM9szs33MmDGDFrgkSZIkNbMiewTvBw6NiAlUkrg5wF/23iEipgJfAWZm5jO97loC/K9eE8ScBlxWYKySJEkq2KZNm+js7OTFF1+sdyhSQxs9ejTjx49n5MiRu/0YhSWCmbk5Ii6mktS1AN/IzIci4gpgWWYuBhYCewPfjQiA32bmrMz8fUT8TyrJJMAVmfn7omKVJElS8To7O3nta1/Lm970Jqqf/SQNUGayfv16Ojs7mTBhwm4/TqFjBDPzh8AP+7Vd3uv2qTs59hvAN4qLTpIkSUPpxRdfNAmU9lBE8PrXv561a9fu0eMUuqC8JEmS1JtJoLTnBuPfkYmgJEmSSuPHP/4xkyZNYuLEiSxYsGC7+2zcuJFzzjmHiRMncswxx/DEE09su+/KK69k4sSJTJo0iSVLluzy+a6//nouvvjiwQp/wL70pS/xwgsv7HrHBlHL+/c3f/M3HHXUURx11FEcdthhtLW1bbuvpaVl232zZs3a5fM98cQTHHHEEYMW/0B1dHSwatWqQh57WCwfIUmSJPXXsaKLhUtWs2ZDN2PbWpk7YxKzp+7+0tJbtmzhYx/7GLfddhvjx4/n6KOPZtasWRx++OF99vv617/Ovvvuy6OPPsqiRYu49NJLufHGG1m1ahWLFi3ioYceYs2aNZx66qn8+te/pqWlZcCxbN68mREjiv8o/qUvfYn3v//9vPrVry78uV5h5U2w9Ap4rhP2GQ+nXA5Tzt7th6v1/fviF7+47faXv/xlVqxYsW27tbWVBx54YLdj6DFU719HRwdnnHHGK17jYLBHsCAdK7o4fsHtTJj3A45fcDsdK16x+oUkSZJ2oGNFF5fd8iBdG7pJoGtDN5fd8uAefaa67777mDhxIocccgijRo1izpw53Hrrra/Y79Zbb+W8884D4L3vfS9Lly4lM7n11luZM2cOr3rVq5gwYQITJ07kvvvue8Xx3/zmNznssMOYNm0ad99997b2888/n49+9KMcc8wxfOYzn+H3v/89s2fPZsqUKbztbW9j5cqVAMyfP58PfOADHHvssRx66KF89atfBSqThMydO5cjjjiCyZMnc+ONNwJw5513csYZZ2x7nosvvpjrr7+eq6++mjVr1jB9+nSmT5++23+33bLyJvj+JfDcU0BWfn//kkr7bqr1/evtO9/5Dueee+6Anmf58uUceeSRHHnkkVx77bXb2q+//npmzZrFySefzCmnnLLT9+OEE07g9NNPZ9KkSXz0ox9l69at2+KZPHkyRxxxBJdeeum2x95777233b755ps5//zz+dnPfsbixYuZO3cuRx11FL/5zW8G9Dp2xR7BAvRcuLo3bQFevnABe/QtliRJUlksXLJ622epHt2btrBwyerd/jzV1dXFgQe+vFT1+PHj+fnPf77T/UaMGME+++zD+vXr6erq4m1ve1uf47u6+iamTz/9NJ///OdZvnw5++yzD9OnT2fq1Knb7u/s7ORnP/sZLS0tfPzjH2fq1Kl0dHRw++2388EPfnBbb9XKlSu59957+dOf/sTUqVM5/fTTueeee3jggQf45S9/ybp16zj66KM54YQTdvh6L7nkEv7hH/6BO+64g/3333+3/ma7bekVsKm7b9um7kr7bvYK1vr+9XjyySd5/PHHOfnkk7e1vfjii7S3tzNixAjmzZvH7NmzX3Hchz70Ia655hpOOOEE5s6d2+e+X/ziF6xcuZL99tuPf/mXf9nh+3HfffexatUqDj74YGbOnMktt9zCcccdx6WXXsry5cvZd999Oe200+jo6NhuDADHHXccs2bN4owzzuC9733vgP5WtbBHsAA7u3BJkiRp19Zs6B5Q+3Dx85//nJNOOokxY8YwatQozjnnnD73/8Vf/MW2UtK77rqLD3zgAwCcfPLJrF+/nj/84Q8AnHnmmbS2trL//vszffp07rvvPu666y7OPfdcWlpaOOCAAzjxxBO5//77GZae6xxYewEWLVrEe9/73j6lu08++STLli3j29/+Np/85Cdf0cu2YcMGNmzYsC2h63l/erzzne9kv/32A9jp+zFt2jQOOeQQWlpaOPfcc7nrrru4//77t50bI0aM4H3vex8//elPi/wT7JSJYAHWbOhm1l53cdeoS3jsVX/JXaMuYdZedw37C5ckSdJwMbatdUDttRg3bhxPPfXUtu3Ozk7GjXtl72Lv/TZv3sxzzz3H61//+pqP35nXvOY1Ne3Xf1bInc0SOWLEiG2lh1Dp9aq7fcYPrL0GA/37L1q06BVloT37H3LIIZx00kl9xg/Wooj3r//9Q/X+mQgW4Ly972PByK8xfq917BUwfq91LBj5Nc7b+5U15JIkSXqluTMm0Tqy7yQsrSNbmDtj0m4/5tFHH80jjzzC448/zksvvcSiRYu2O3PkrFmzuOGGG4DKeK2TTz6ZiGDWrFksWrSIjRs38vjjj/PII48wbdq0Pscec8wx/OQnP2H9+vVs2rSJ7373uzuM5x3veAff+ta3gMq4sv3335/Xve51QGWc4osvvsj69eu58847Ofroo3nHO97BjTfeyJYtW1i7di0//elPmTZtGgcffDCrVq1i48aNbNiwgaVLl257jte+9rX88Y9/3O2/2W475XIY2S9pH9laad9Ntb5/AL/61a949tlnOfbYY7e1Pfvss2zcuBGAdevWcffdd79iEpa2tjba2tq46667ALa9P9uzo/cDKqWhjz/+OFu3buXGG2/k7W9/O9OmTeMnP/kJ69atY8uWLXznO9/hxBNPBOCAAw7g4YcfZuvWrXzve9/b9hxFvn+OESzAZ0beyKs3v9Sn7dXxEp8ZeSPw/9YnKEmSpAbSMw5wMGcNHTFiBNdccw0zZsxgy5YtXHDBBbz1rW8F4PLLL6e9vZ1Zs2Zx4YUX8oEPfICJEyey3377sWjRIgDe+ta3cvbZZ3P44YczYsQIrr322lfMGPrGN76R+fPnc+yxx9LW1sZRRx21w3jmz5/PBRdcwJQpU3j1q1+9LfkEmDJlCtOnT2fdunV87nOfY+zYsbznPe/hnnvu4cgjjyQiuOqqq3jDG94AwNlnn80RRxzBhAkT+oxJvOiii5g5cyZjx47ljjvu2O2/3YD1jAMcxFlDa33/oNIbOGfOnD49bQ8//DAf+chH2Guvvdi6dSvz5s3b7myc3/zmN7nggguICE477bQdxrOj9+NXv/oVRx99NBdffDGPPvoo06dP5z3veQ977bUXCxYsYPr06WQmp59+OmeeeSYACxYs4IwzzmDMmDG0t7fz/PPPAzBnzhz+6q/+iquvvpqbb76ZN7/5zbv99+svMnPQHqye2tvbc9myZfUOo2J+G7C9v2vA/A1DHY0kSdKw8PDDD/OWt7yl3mEMe/Pnz2fvvffm05/+dL1D0W648847+cIXvsC//uu/Fvo82/v3FBHLM7O9luMtDS1CATXRkiRJkjRYLA0twimXV9ZJ6T1l7h7WREuSJKkc5s+fX+8QtAdOOukkTjrppHqHsUv2CBZhytnw7qthnwOBqPx+99V7VBMtSZIkSYPFHsGiTDnbxE+SJKmfzNzlVPqSdm4w5nmxR1CSJElDYvTo0axfv35QPsRKZZWZrF+/ntGjR+/R49gjKEmSpCExfvx4Ojs7Wbt2bb1DkRra6NGjGT9+zyaiNBGUJEnSkBg5ciQTJkyodxiSsDRUkiRJkkrHRFCSJEmSSsZEUJIkSZJKJppl1qaIWAs8We84tmN/YF29g1BT8xxTkTy/VCTPLxXJ80tFGq7n18GZOaaWHZsmERyuImJZZrbXOw41L88xFcnzS0Xy/FKRPL9UpGY4vywNlSRJkqSSMRGUJEmSpJIxESzedfUOQE3Pc0xF8vxSkTy/VCTPLxWp4c8vxwhKkiRJUsnYIyhJkiRJJWMiWKCImBkRqyPi0YiYV+941Fwi4omIeDAiHoiIZfWOR40vIr4REc9ExH/0atsvIm6LiEeqv/etZ4xqXDs4v+ZHRFf1OvZARPzXesaoxhURB0bEHRGxKiIeiohPVNu9hmmP7eT8auhrmKWhBYmIFuDXwDuBTuB+4NzMXFXXwNQ0IuIJoD0zh+MaNmpAEXEC8Dzwz5l5RLXtKuD3mbmg+oXWvpl5aT3jVGPawfk1H3g+M79Qz9jU+CLijcAbM/MXEfFaYDkwGzgfr2HaQzs5v86mga9h9ggWZxrwhEfNyQAAA8xJREFUaGY+lpkvAYuAM+sckyTtUGb+FPh9v+YzgRuqt2+g8h+fNGA7OL+kQZGZT2fmL6q3/wg8DIzDa5gGwU7Or4ZmIlicccBTvbY7aYITRsNKAv8WEcsj4qJ6B6OmdUBmPl29/Z/AAfUMRk3p4ohYWS0dtWxPeywi3gRMBX6O1zANsn7nFzTwNcxEUGpcb8/MPwfeBXysWnYlFSYrYwkcT6DB9E/Am4GjgKeB/13fcNToImJv4F+AT2bmH3rf5zVMe2o751dDX8NMBIvTBRzYa3t8tU0aFJnZVf39DPA9KuXI0mD7XXVsRM8YiWfqHI+aSGb+LjO3ZOZW4Kt4HdMeiIiRVD6kfyszb6k2ew3ToNje+dXo1zATweLcDxwaERMiYhQwB1hc55jUJCLiNdXBykTEa4DTgP/Y+VHSblkMnFe9fR5wax1jUZPp+YBe9R68jmk3RUQAXwcezsx/6HWX1zDtsR2dX41+DXPW0AJVp5D9EtACfCMz/77OIalJRMQhVHoBAUYA3/b80p6KiO8AJwH7A78DPg90ADcBBwFPAmdnphN+aMB2cH6dRKWkKoEngI/0Gs8l1Swi3g78O/AgsLXa/D+ojOPyGqY9spPz61wa+BpmIihJkiRJJWNpqCRJkiSVjImgJEmSJJWMiaAkSZIklYyJoCRJkiSVjImgJEmSJJWMiaAkSf1ExJaIeKDXz7xBfOw3RURDrTUlSWo+I+odgCRJw1B3Zh5V7yAkSSqKPYKSJNUoIp6IiKsi4sGIuC8iJlbb3xQRt0fEyohYGhEHVdsPiIjvRcQvqz/HVR+qJSK+GhEPRcS/RURr3V6UJKmUTAQlSXql1n6loef0uu+5zJwMXAN8qdr2ZeCGzJwCfAu4utp+NfCTzDwS+HPgoWr7ocC1mflWYAPw3wp+PZIk9RGZWe8YJEkaViLi+czcezvtTwAnZ+ZjETES+M/MfH1ErAPemJmbqu1PZ+b+EbEWGJ+ZG3s9xpuA2zLz0Or2pcDIzPy74l+ZJEkV9ghKkjQwuYPbA7Gx1+0tOGZfkjTETAQlSRqYc3r9vqd6+2fAnOrt9wH/Xr29FPhrgIhoiYh9hipISZJ2xm8gJUl6pdaIeKDX9o8zs2cJiX0jYiWVXr1zq20fB74ZEXOBtcCHqu2fAK6LiAup9Pz9NfB04dFLkrQLjhGUJKlG1TGC7Zm5rt6xSJK0JywNlSRJkqSSsUdQkiRJkkrGHkFJkiRJKhkTQUmSJEkqGRNBSZIkSSoZE0FJkiRJKhkTQUmSJEkqGRNBSZIkSSqZ/x/Zo0//DFJCBQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x1080 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot train and validation accuracies of the two models\n",
    "\n",
    "train_accs = []\n",
    "val_accs = []\n",
    "for dropout in dropout_choices:\n",
    "  solver = solvers[dropout]\n",
    "  train_accs.append(solver.train_acc_history[-1])\n",
    "  val_accs.append(solver.val_acc_history[-1])\n",
    "\n",
    "plt.subplot(3, 1, 1)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].train_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Train accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "  \n",
    "plt.subplot(3, 1, 2)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].val_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Val accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "\n",
    "plt.gcf().set_size_inches(15, 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Question\n",
    "Explain what you see in this experiment. What does it suggest about dropout?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Answer\n",
    "Dropout makes training a little slower, but it slightly increases the performance for validation set. This shows that it helps reduce overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
